Support --pip-version 24.1 and Python 3.13.
#2435
Conversation
Pex has been testing against Python 3.13 alpha and beta releases for a while now using a patched Pip with Python 3.13 support. Those patches have made it into the Pip 24.1 release; so now Pex can officially support Python 3.13. The Pip 24.1 release notes: https://pip.pypa.io/en/stable/news/#v24-1 Closes pex-tool#2406
| @attr.s(frozen=True) | ||
| class _Issue10050Analyzer(ErrorAnalyzer): | ||
| # Part of the workaround for: https://github.com/pypa/pip/issues/10050 | ||
| class EvaluationEnvironment(dict): |
There was a problem hiding this comment.
As noted in the locker patches, pypa/packaging#802, which Pip 24.1 uses, necessitated a different tactic here. Instead of scraping logs for ~KeyErrors when a marker key was missing, we must appease the packaging "local" hack for the python_full_version marker by always returning a stringy value that supports .endswith. We do this here and then just test values in the now-patched markers._eval_op below in pex/pip/foreign_platform/markers.py: https://github.com/pex-tool/pex/pull/2435/files#diff-20427b0b02256628895cffcee477785698b69ef0f4208176e6638e76f1279bbdR45
This switch in tactic mirrors the locker patching tactic already in place, patching both markers and the eval of them.
| @@ -101,7 +101,7 @@ def test_lock_uncompilable_sdist( | |||
|
|
|||
|
|
|||
| setup_kwargs = dict( | |||
| name="pex.tests.bad-c-extension", | |||
| name="pex_tests_bad_c_extension", | |||
There was a problem hiding this comment.
This was the easiest way to deal with changes in how Pip, vendored setuptools and vendored packaging handle name normalization: just use a pre-normalized name, which skates through all combos of the above as a noop.
| complete_platform_data = json.loads( | ||
| subprocess.check_output( | ||
| args=docker_run_args + ["interpreter", "inspect", "--markers", "--tags"] | ||
| ) | ||
| ) | ||
| container_interpreter = CompletePlatform.create( | ||
| marker_environment=MarkerEnvironment(**complete_platform_data["marker_environment"]), | ||
| supported_tags=CompatibilityTags.from_strings(complete_platform_data["compatible_tags"]), | ||
| ) | ||
| pip_version = try_( | ||
| compatible_version( | ||
| targets=Targets( | ||
| interpreters=tuple([PythonInterpreter.get()]), | ||
| complete_platforms=tuple([container_interpreter]), | ||
| ), | ||
| requested_version=PipVersion.DEFAULT, | ||
| context=__name__ + ".test_ssl_context", | ||
| ) | ||
| ) |
There was a problem hiding this comment.
N.B.: We need the lock created in the container to use a Pip version compatible with the outer test's interpreter. Previously, the one case where the default did not satisfy this constraint was handled by plumbing --env _PEX_PIP_VERSION. Now, we calculate a compatible version up front instead and plumb that to the lock explicitly.
| ) | ||
|
|
||
| VENDORED = v20_3_4_patched | ||
| LATEST = LatestPipVersion() | ||
| DEFAULT = DefaultPipVersion(preferred=(VENDORED, v23_2)) | ||
| DEFAULT = DefaultPipVersion(preferred=(VENDORED, v23_2, v24_1)) |
There was a problem hiding this comment.
Just to confirm by default pex will select the first compatible pip version right?
There was a problem hiding this comment.
That's right. These entries effectively mark where Python broke Pip. In this case, Pip relied on APIs in importlib that Python yanked in 3.13; so Pip 24.1 is the first Pip version to support Python>=3.13. So that is the default for Python>=3.13 until some new Python no longer works against Pip 24.1 in which case I add a new entry. The Pip 23.2 marks the transition of Pip to Python 3.12 support where Python yanked distutils, breaking Pip<23.2.
Agreed. One thing I started on in 2020 but never finished since its somewhat frivolous and self-defeating, is a massive re-structure of Pex modules to clearly delineate code needed at PEX runtime from code needed by the Pex tool at build time. The vendored Pip is only in the latter set and does not get included in a PEX files So its worse than you thought, but required to ensure the PEX can boot under various pythons and not choke on syntax errors in the |
Pex has been testing against Python 3.13 alpha and then beta releases
for a while now using a patched Pip with Python 3.13 support. Those
patches have made it into the Pip 24.1 release; so now Pex can
officially support Python 3.13.
The Pip 24.1 release notes: https://pip.pypa.io/en/stable/news/#v24-1
Closes #2406